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I.  BACKGROUND 


Accurate  cost  estimation  of  alternative  weapons  systems 
is  essential  for  long  range  military  planning  and  budgeting 
decisions.  The  choice  among  competing  systems  is  based  on 
trade-offs  between  performance  parameters  and  mission  require¬ 
ments  with  an  ever  increasing  importance  being  placed  on  cost. 

In  fact,  with  the  issuing  of  Department  of  Defense  (DoD) 
Instruction  5000.1,  cost  has  been  upgraded  to  a  principle 
design  parameter.  That  directive  defines  specific  "Design  to 
Cost"  policies  during  requirements  formulation  in  determining 
which  system  is  most  cost  effective. 

Traditionally,  weapon  system  cost  estimates  have  been 
prepared  using  Industrial  Engineering  (I.E.)  techniques. 

These  techniques  involved  detailed  studies  of  the  operations 
and  materials  required  to  produce  the  new  system.  The  cost 
estimate  frequently  required  several  thousand  hours  to  pro¬ 
duce  with  volumes  of  supporting  documentation.  Small  changes 
in  design  often  necessitate  extensive  revision  of  these  esti¬ 
mates.  In  spite  of  all  the  time  and  effort  involved  in  pre¬ 
paring  these  estimates,  their  accuracy  leaves  much  to  be 
desired.  This  is  evidenced  by  the  large  cost  overuns  cited 
by  the  annual  General  Accounting  Office  (GAO)  reports  to 
Congress.  In  1972,  for  example,  the  GAO  reported  that  the  Navy 
had  experienced  a  cost  growth  of  $19  billion  on  24  weapon  sys¬ 
tems  in  FY  1971.  Approximately  15%  of  this  cost  growth  was 
attributed  to  poor  initial  cost  estimates  for  the  weapon  systems. 
The  report  went  on  to  make  the  following  recommendation: 

"Develop  and  implement  DoD  wide  guidance  for  consistent 
and  effective'  cost  estimating  procedures  and  practices 
particularly  with  regard  to,  ...  an  effective  indepen¬ 
dent  review  of  cost  estimates." 

This  report  and  other  high  level  directives  implementing 
its  recommendations  have  resulted  in  increased  interest  and 
reliance  on  a  statistical  approach  called  parametric  cost 
estimation  defined  by  Baker  (2)  as 

"An  estimate  which  predicts  costs  by  means  of  explanatory 
variables  such  as  performance  characteristics ,  physical 
characteristics,  and  characteristics  relevant  to  the 
development  process,  as  derived  from  experience  on 
logically  related  systems." 

The  construction  and  use  of  "cost  estimating  relationships" 
(CER's)  is  the  basis  for  making  independent  parametric  cost 
estimates.  CER's  are  mathematical  equations  which  relate  sys¬ 
tem  costs  to  various  explanatory  variables.  They  are  based  on 


the  premise  that  the  cost  of  a  weapons  sytem  is  logically 
related  in  a  quantifiable  way  to  the  system's  physical  and 
performance  characteristics.  They  are  most  generally 
derived  through  statistical  regression  analysis  of  historical 
cost  data.  These  techniques  are  described  in  (14) .  Some 
examples  of  their  use  appear  in  ( 9  ) ,  (10) ,  (15)  ,  and  (17) . 

The  parametric  approach  has  some  distinct  advantages 
and  disadvantages  compared  to  I.E.  methodology.  On  the 
plus  side  are: 

1.  Parametric  cost  estimates  can  be  developed  during 
the  concept  formulation  stage  of  the  acquisition 
process  before  detailed  engineering  plans  are 
available.  These  early  cost  estimates  can  be 
used  to: 

(a)  Identify  possible  cost /performance  trade¬ 
offs  in  the  design  effort. 

(b)  Provide  a  basis  for  cost/effectiveness 
review  of  performance  specifications. 

(c)  Provide  information  useful  in  the  ranking 
of  competing  alternatives. 

(d)  Suggest  a  need  for  identifying  and 
considering  new  alternatives. 

2.  Historical  cost  data  incorporates  system  development 
setbacks  such  as  engineering  and  design  specification 
changes  and  other  items  that  are  not  identifiable  at 
the  time  of  design.  Industrial  engineering  estimates 
tend  to  be  optimistic  in  that  they  don't  allow  for 
unforeseen  problems.  Unexpected  engineering  or 
design  changes  usually  bring  about  unexpected  in¬ 
creases  in  system  cost.  Cost  estimating  relationships 
based  on  historical  data  will  incorporate  some  of 
these  unknowns  into  the  cost  estimate. 

Along  with  the  cited  advantages  of  parametric  cost  esti¬ 
mation,  there  are  some  troublesome  aspects.  Assumptions, 
subjective  assessments  and  choice  of  methodolgy  can  lead 
different  analysts  to  different  estimates.  This  kind  of 
ambiguity  seems  unavoidable  in  empirical  model  building 
studies  and  is  the  basis  for  some  critcism.  While  there  are 
general  guidelines  as  to  what  constitutes  sound  statistical 
practice,  there  is  rarely  a  universally  accepted  "best" 
approach  to  modeling  and  prediction. 
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This  paper  focuses  on  the  role  of  analogy  in  empirical 
model  building.  A  measure  of  analogy  between  systems  of  the 
same  general  type  is  proposed  and  its  properties  studied. 

A  new  criteria  for  variable  selection  based  on  this  measure 
is  explored  and  a  parametric  cost  estimation  example  is 
given . 


II.  THE  ROLE  OF  ANALOGY  IN  PREDICTION 

The  phrase  "logically  related  system"  in  the  cited 
definition  of  parametric  cost  estimation  is  subject  to  all 
kinds  of  interpretation  and  degrees  of  relation.  Certainly 
there  is  no  historical  system  identical  in  all  respects  to 
the  object  system  (the  system  whose  cost  we  wish  to  predict) 
else  the  problem  would  not  exist.  At  the  other  extreme,  all 
military  systems  are  "logically  related"  in  (at  least)  the 
sense  that  they  are  military  systems.  Message  carrying 
pidgeons,  air-to-air  missiles,  jet  aircraft  and  frizbees  are 
"logically  related"  in  that  they  all  fly.  Obviously,  the 
analyst  must  take  into  account  the  degree  of  analogy  between 
each  system  (which  is  a  candidate  for  the  historical  data 
base)  and  the  object  system.  Analogy,  according  to  Webster, 
is  "a  partial  similarity  between  like  features  of  two  things 
on  which  a  comparison  may  be  based."  How  does  one  measure 
the  degree  of  analogy  between  "logically  related"  systems  and 
how  can  one  exploit  these  partial  similarities  in  predicting 
the  cost  of  an  object  system? 

Having  gathered  and  adjusted  historical  data  on  systems 
judged  more-or-less  analogous  to  a  proposed  system  whose  cost 
is  to  be  estimated,  the  analyst  proceeds  with  the  task  of 
developing  a  "best"  CER.  This  involves  selecting  the  form 
of  the  CER,  deciding  which  of  the  system  variables  (performance 
characteristics,  design  specifications,  etc.)  to  include  as 
predictor  variables,  and  assessing  the  precision  of  the 
estimate.  In  parametric  cost  estimation,  this  is  usually  done 
through  the  use  of  multiple  regression  and  some  standard 
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variable  selection  criterion  such  as  maximizing  adjusted  R 
(which  is  equivalent  to  minimizing  mean  square  error  [MSE]), 
maximizing  F,  using  Mallow's  C  ,  etc. 

Sr 

All  of  these  techniques  share  two  properties:  (1)  For 
any  fixed  number  of  variables  in  the  prediction  equation,  the 
optimal  set  of  variables  is  that  set  which  minimizes  the  MSE; 

(2)  They  all  ignore  the  values  of  the  variables  of  the  system 
whose  cost  is  being  estimated.  The  first  of  these  properties 
is  reasonable  but  myopic  when  the  object  is  prediction.  The 
second  property  seems  contrary  to  common  sense. 
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Suppose  there  are  n  systems  in  the  historical  data  base. 

Associated  with  the  i^1  such  system  is  a  cost  Y^  and  values  of 

p  (candidate)  predictor  variables  X. -,..., X. X.  .  Let 

”  ™  1  ^P 


Y'  =  (Y1...Yi...Yn)  and  XI  =  (X^, 


•*Xij*“Xnj)  '  j 


1,2,. 


denote  these  historical  costs  and  system  characteristics. 
Furthermore,  let 


X. 

J 


i  n 

1  l 

n  ,.i. 


and 


k=l 


Sij 


i  n  _  _ 

l  (X..-X. )  (X..-X.) 
n-1  ki  i  k]  ] 


denote  the  sample  means  and  covariances.  Denoting  the  values 
of  the  proposed  system  by  lower  case  letters ,  we  wish  to  pre¬ 
dict  its  cost  y  by  exploiting  the  predictive  ability  its 
characteristics  X’  =  (x^, . . . ,Xj , . . . ,x^) .  This  predictive 

ability  is  inferred  from  the  apparent  relation  between  historical 
costs  and  characteristics  and  the  degree  of  analogy  between  the 
proposed  system  and  these  historical  data.  How  analogous  is 
this  proposed  system  to  the  historical  data? 


(a)  Marginal  comparisons:  Analogy  on  a  single  dimension 
is  straightforward.  A  statistic  commonly  used  as  a  non¬ 
negative  distance  index  is  simply  the  square  of  the  standardized 
distance  between  x^  and  the  mean  of  the  X^j's,  namely 


M 


j 


2 


where  s^  is  defined  as  (Sjj)^ 
indicate  a  low  degree  of  analogy. 


Large  values  of  this  statistic 


(b)  High  dimensional  comparisons:  The  collection  of 
marginal  indices  {M^,M2 , . . . ,M  }  can  give  a  very  misleading 

impression  of  the  overall  degree  of  analogy.  Even  when  Mj  is 

small  for  every  j ,  the  proposed  system  can  be  terribly  non- 
analogous  to  the  historical  data.  A  simple  bivariate  example 
will  illustrate  this  assertion.  Suppose  X^  and  X2  denote 

weight  and  maneuverability,  respectively,  and  that  x^  and  x2 

are  each  within  one  standard  deviation  of  their  respective 
means ,  i . e . , 


»  M2  <  1. 
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Suppose  further  that,  historically,  heavy  systems  tended  to 
be  less  maneuverable,  i.e.  ,  p  <0,  but  the  proposed 

X1X2 

system  is  a  little  heavier  and  slightly  more  maneuverable 
than  the  average.  The  situation  is  depicted  in 

Figure  1 


We  see  that  is  marginally  analogous  to  the  historical 

data  on  both  weight  and  maneuverability  but  not  at  all 
analogous  when  viewed  in  two  dimensions.  Comparing  (x^x^ 

to  (X^,X2>  marginally  ignores  important  relational  information. 

The  geometry  in  higher  dimensional  spaces  is  more  difficult  to 
grasp. 

A  measure  of  analogy  which  incorporates  relational 
information  was  suggested  in  1930  by  P.C.  Mahalanobis  (12). 

He  proposed 

m  -  r1(ia’-j£<2)) 

as  a  measure  of  the  distance  between  two  multivariate  popu¬ 
lations  with  mean  vectors  jj^  and  jj  ^  ,  respectively,  and 
common  covariance  matrix  £.  Replacing  the  parameter  values 
by  estimates,  we  obtain  (in  our  notation) 

M  =  (x-X)  ’  S“1(x-X) 

where  S  *  (s^j)  an<*  =  (X^,...,X  ).  Except  for  a  multi¬ 
plicative  constant,  this  is  Hotelling's  T2  statistic  used 
to  test  that  x  and  the  historical  data  came  from  the  same 
population.  In  the  previous  bivariate  example,  it  is  easy 
to  show  that 
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M  =  [M1-2p(M1M2)J5+M2]  , 


which  can  be  arbitrarily  large  even  when  and  M2  are  small. 
For  example,  with  =  M2  »  e , 

M  =  —  and  ^  lim  M  =  «. 

1+P  p-»—  1 

In  what  follows ,  we  propose  Mahalanobis  distance  M  as  a 
measure  of  analogy  and  discuss  its  implications  in  the  process 
of  tailoring  a  CER  to  a  specific  object  system  for  the  purpose 
of  predicting  that  system's  cost.  This  is  a  distinct  depar¬ 
ture  from  standard  procedures  recommended  ( 14) ,  ( 15)  ana  used 
in  developing  every  CER  with  which  the  authors  are  familiar. 

The  distinction  is  fundamental  and  goes  beyond  measure  of 
analogy.  The  standard  approach  appears  more  oriented  toward 
developing  a  cost  explaining  equation  relating  costs  of  a 
class  (e.g.,  sonars,  airframes,  tanks,  etc.)  of  historical 
systems  to  the  characteristics  of  those  systems.  One  need  not 
have  any  specific  object  system  in  mind  while  developing  such 
a  general  purpose  descriptive  equation.  In  fact,  armed  with 
an  airframe  CER  based  on  the  explanatory  variable  "weight" ,  two 
radically  different  airframes  of  the  same  weight  would  be 
estimated  to  cost  the  same  amount.  Hallows  (13)  defined  six 
potential  uses  of  a  regression  equation  which  include  (a)  pure 
description  and  (b)  prediction.  Lindley  (11)  emphasizes  that 
the  technique  used  to  develop  a  regression  equation  ought  to 
be  related  to  the  intended  use.  In  the  present  context,  the 
intended  use  is  prediction  of  the  cost  of  a  specific  system  so 
that  using  a  CER  (which  was  developed  to  describe  historical 
relations  without  reference  to  any  specific  object  system) 
to  predict  cost  of  a  specific  system  is  contrary  to  Lindley 's 
recommendation  and  common  sense. 


III.  EXPLOITING  ANALOGY  IN  DEVELOPING 
MODELS  FOR  PREDICTION 

As  mentioned  in  the  previous  section,  most  standard 
variable  selection  techniques  share  the  property  that,  for 
any  given  number  of  variables  in  the  regression  function,  the 
optimal  set  is  that  set  which  minimizes  residual  mean  square 
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error  (or,  equivalently,  maximizes  r  ) .  The  objective  system 
may  be  rather  nonanalogous  to  the  historical  data  (large  M) 
when  we  consider  the  subset  of  variables  identified  as 
"optimal"  by  the  criteria  used  to  develope  the  CER.  Often, 
there  are  several  k-variable  models  which  come  close  to  the 
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"optimum"  in  terms  of  r  and  other  measures  of  model  aptness 
based  on  residual  analysis.  In  these  cases,  by  using  a 
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slightly  suboptimal  set  of  prediction  variables  (slight 

.  2 

decrease  m  r  )  it  may  be  possible  to  substantially  improve 
the  degree  of  analogy  (decrease  in  M  ) .  What  is  the  role 
of  analogy  in  prediction  and  how  can  one  evaluate  the  trade¬ 
off  of  fit  for  analogy? 

The  width  of  the  prediction  interval  at  the  point  corres¬ 
ponding  to  the  objective  system  is  a  numeraire  which  seems  like 
a  reasonable  basis  for  choosing  between  alternative  models.  We 
shall  consider  a  monotone  function  of  the  width  for  simplicity, 
namely,  the  square  of  the  half-width,  viz. 


w  =  p  .  *MSE*A  +  *—)  , 

l-f;l,n-k-l  n_1  n 

where  F  is  the  (1-^)^  fractile  of  an  F  distribution 

l-~;l,n-k-l  * 


with  1  and  n-k-1  degrees  of  freedom.  This  measure  W  combines 
"fit"  (MSE)  and  "degree  of  analogy"  (M)  with  a  factor  F  which 
penalizes  for  using  too  many  variables  (increasing  k)  or  ex¬ 
cluding  points  from  the  data  base  (decreasing  n) .  In  this 
form,  the  role  of  analogy,  as  measured  by  Mahalanobis^distance, 

is  evident.  It  enters  as  a  term  in  the  multiplier  (— +  n+-^~) 

n- 1  n 

of  MSE.  Failure  to  consider  this  factor  in  selecting  a  CER 
could  have  a  marked  effect  on  predictor  precision  as  measured 
by  prediction  interval  width.  We  hasten  to  point  out  that  the 
W-criterion  is  not  being  suggested  as  a  universally  "best" 
variable  selection  procedure.  Rather,  it  should  be  viewed  as  a 

device  to  reduce  the  2^  candidate  models  to  a  more  manageable 
number  so  that  detailed  residual  analysis  is  economically 
feasible.  Models  which  appear  attractive  under  criteria  based 
2 

on  maximizing  R  but  are  far  from  W-optimal  obviously  involve 
relatively  large  extrapolation  at  the  point  corresponding  to  the 
object  system.  With  extrapolation  comes  concern  that  the  model 
which  seems  to  fit  the  historical  data  best  may  not  be  valid  at 
the  point  under  prediction. 


IV.  PROPERTIES  OF  W 

While  the  quantity  W  is  related  to  a  100(l-a)%  prediction 

interval,  it  is  not  true  that  y(x)  ±  w*5  covers  y(x)  with 
probability  1-a  when  the  same  data  is  used  to  select  the  model 
and  compute  the  prediction  interval.  The  confidence  coefficient 
resulting  from  using  the  model  with  the  smallest  W  will,  in 
general,  be  less  than  1-a.  This  drawback  is  not  limited  to  the 
W-criterion  but  is  a  general  problem  associated  with  empirical 
model  building  and  subsequent  inference  using  the  same  data. 
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It  is  well  known  (for  example,  see  (  5)  and  (16))  that  choosing 

2 

the  k-variable  model  which  maximizes  R  has  an  inflationary 
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effect  on  the  distribution  of  R  ,  invalidating  the  usual  signi¬ 
ficance  tests.  Distribution- free  exact  tests  have  recently  been 
developed  by  Edwards  and  Wallenius  ( 5  )  to  deal  with  this 
problem.  Distributional  properties  of  W  are  not  well  under¬ 
stood  unless  model  building  and  inference  are  made  on  independent 
data  sets ,  a  luxury  of  sample  size  not  commonly  encountered  in 
parametric  cost  estimation.  A  monotonicity  result  which  is 
useful  in  variable  selection  algorithms  and  decisions  concerning 
inclusion  or  exclusion  of  points  in  the  historical  data  base  is 
given  in  the  following 


THEOREM*:  Let  denote  the  Mahalanobis  distance  between 

?k 


x;.  *  (x^,...,^)  and  =  (Xlf...,X^)  for  k  =»  l,...,p  and  let 

Vi'V 


Then  A j 


(xi-fl~xi+l) 

S22(H’ 


where 


rj  *  S21S11S12/S22'  xj+l  Xj+l+S21Sll(xj_?j) 


and  S  = 


S11  S12 
S21  S22 


is  the  partitioned  covariance  matrix  of 


X1,X2,. . . ,X^+1  with  S22 


being  a  scalar. 


A  proof  of  the  theorem  is  given  in  (  8 ) .  Since  0  £  r  <1 
and  S22  is  the  sample  variance  of  X^+1,  it  follows  that  _>  0 

with  equality  if,  and  only  if,  t^lat  is*  if  Xj+1  ^es 

on  the  hyperplane  defined  by  the  regression  of  on  the 

variables  (X^,...,X. ).  Thus,  augmenting  a  specific  j-variable 

cannot  decrease  Mahalanobis  distance  (i.e.  ,  increase  analogy)  so 
that,  using  the  W-criterion,  a  variable  will  be  added  to  a  given 
model  only  if,  by  so  doing,  the  decrease  in  MSE  is  large  enough 
to  offset  increases  in  M  and  F.  This  observation  explains  why 
W-optimal  models  tend  to  be  more  parsimonious  than  MSE  optimal 
models,  a  desireable  property  in  view  of  the  commonly  held 
opinion  among  satisticians  that  the  minimum  MSE  criterion 
frequently  results  in  overfitting. 


Besides  providing  this  kind  of  insight  into  the  effect  on 
W  of  adding  an  explanatory  variable  to  a  given  model,  the  mono¬ 
tonicity  theorem  is  useful  in  developing  a  branch  and  bound 
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algorithm  similar  to  that  of  Fumival  (  6  )  which  makes  it 
possible  to  efficiently  identify  models  with  small  W's 

without  computing  all  2^-1  possible  regressions.  These 
computational  aspects  are  discussed  in  ( 8  )  along  with  a 
treatment  of  various  suboptimal  search  algorithms  (e.g. , 
forward  selection,  backward  selection,  etc.) 

The  monotonicity  theorem  is  also  useful  in  dealing  with 
questions  about  setting  aside  points  in  the  historical  data 
base.  It  is  quite  plausible  that  inclusion  of  one  or  more 
systems  in  the  data  base  which  are  rather  nonanalogous  to  the 
object  system  and/or  the  other  reference  systems  could  have 
a  detrimental  effect  on  prediction.  The  influence  of  indi¬ 
vidual  observations  or  groups  of  observations  on  various 
quantities  of  interest  in  a  statistical  analysis  has  received 
considerable  attention  in  recent  literature  (3),  (4),  (7), 
etc.  Most  of  this  work  is  concerned  with  identifying  outliers 
ai.d  assessing  the  effect  of  setting  them  aside  on  estimates  of 
model  parameters.  Since  the  focus  of  our  work  is  on  pre¬ 
diction,  these  earlier  studies  are  not  directly  applicable. 
Space  limitations  here  do  not  allow  for  treatment  of  questions 
on  the  role  of  analogy  in  choosing  the  data  base.  The 
interested  reader  is  referred  to  (  8  )  for  further  information. 


V.  APPLICATION 

Data  that  formed  the  basis  for  an  actual  parametric  cost 
estimation  study  (18)  was  used  to  compare  the  performance  of 
the  W-criterion  with  other  standard  and  not- so- standard  variable 
selection  techniques.  The  data  consisted  of  costs  and  12  design/ 
performance  characteristics  of  23  single  engine  jet  interceptors 
built  between  1947  and  1969.  The  object  of  the  study  was  to 
predict  costs  of  the  F-14  and  F-15  aircraft.  As  is  common 
practice,  see  (10) ,  (14) ,  and  (15)  for  example,  the  data  were 
log- transformed  prior  to  analysis.  (Our  analysis  confirmed 
that  regressions  were  more  nearly  linear  and  residuals  better 
behaved  using  the  transformed  data.)  Cost  was  defined  as  g 
"total  flyaway  cost  (adjusted  to  1972  constant  dollars  x  10  ) 

of  the  100th  production  aircraft."  The  candidate  predictor 
variables  were 

X^  *  Wing  Loading  Ratio 

Xj  =  Aspect  Ratio 

X^  -  Full  to  Empty  Weight  Ratio 

X^  ■  Thickness-to-Cord  Ratio 

Xg  ■  Lift  to  Drag  Ratio 

Xg  -  Total  Avionics  Input  Power  in  kva 

X^  ■  Maximum  Speed  in  knots  (Clean,  Combat  Weight) 

Xg  *  Weight  Empty  in  lbs 
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Xg  =  Rate  of  Climb  in  ft/min,  (sea  level,  combat  weight 
and  power) 

X^g  =  Combat  Ceiling  in  feet 

X^  =  Ferry  Range  in  nautical  miles 

x^2  =  Sea  Level  Static  Thrust  (max)  in  lbs. 

The  criteria  compared  were  Minimum  MSE ,  Minimum  C  , 

2  1  * 

Maximum  F,  Maximum  R  (subject  to  a  1%  "elbow  rule"  ) ,  Minimum 

Z 

Mean  Square  Error  of  Prediction  (MSEP  ) ,  and  Minimum  W  (nominal 
95%  prediction  interval) .  The  procedure  followed  consisted  of 
setting  aside  each  aircraft,  finding  the  best  model  under  each 
of  the  6  criteria  based  on  the  remaining  22  data  points  and 
predicting  the  cost  of  the  deleted  aircraft  using  these  "optimal" 
models.  While  selecting  a  "best"  CER  based  on  any  single  vari¬ 
able  selection  criterion  is  a  myopic  data  analytic  practice, 
it  is  a  necessary  evil  if  the  relative  performance  of  different 
criteria  are  to  be  compared  over  a  given  data  set.  Absolute 
errors  (differences  between  actual  cost  and  predicted  cost)  and 
Minimum  W  gain  (difference  in  Minimum  W  absolute  error  and  the 
absolute  errors  of  other  optimal  models)  were  computed  and  are 
displayed  in  Table  1.  A  gain  denoted  by  "0"  indicates  the 
optimal  model  selected  by  a  given  criterion  coincided  with  the 
Minimum  W  model  while  "00"  indicates  different  models  but  a 
gain  of  less  than  .5  x  10“2. 


1  2 

Since  R  is  nondecreasing  as  variables  are  added  to  the 
prediction  equation,  a  stopping  rule  is  needed.  It  is  rather 
common  to  stop  adding  variables  when  the  marginal  increase  in 
2 

R  is  small.  We  defined  "small"  to  mean  "no  more  than  1% 

2 

increase  in  R  could  be  achieved  by  adding  more  variables." 

2 

Minimum  MSEP,  proposed  in  reference  (  1) ,  is  the  only 
other  selection  criterion  besides  minimum  W  which  takes  into 
account  the  point  at  which  prediction  is  to  be  made.  While 
the  concept  seems  reasonable  at  first  glance,  its  successful 
application  requires  a  good  estimate  of  the  bias  in  each 
submodel.  This  logically  requires  knowledge  (or  a  good  estimate) 
of  the  true  E(y|x).  But  if  this  quantity  were  known,  the 
prediction  problem  would  be  solved.  The  results  of  numerical 
work  confirms  our  theoretical  skepticism  of  Allen's  approach. 
Detailed  criticism  can  be  found  in  (  8 ) . 
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In  the  course  of  the  analysis,  95%  prediction  intervals 
were  calculated  for  each  of  the  23  costs  based  on  the  models 
selected  by  the  various  criteria.  Their  average  widths  and 
actual  coverage  frequencies  are  given  in 


Table  2.  Average  Nominal  95%  Prediction  Interval  Widths  and 
Frequency  of  Coverage  in  Logarithmic  Units 


Avrg.  Width 
Coverage 


Min  MSE  Min  C^ 

1.385  1.250 

86.96  91.30 


Max  F  R2 

1.778  1.287 

91.30  91.30 


MSEP  Min  W 

2.288  1.223 

95.65  95.65 


Examining  tables  1  and  2,  we  note  that  the  minimum  W 
criterion  had  the  smallest  mean  absolute  error  of  all  criteria 
tested.  By  design,  it  has  the  shortest  prediction  interval 
width  and,  quite  surprisingly,  was  unsurpassed  in  the  fre¬ 
quency  with  which  it  covered  the  actual  cost.  At  least  for 
this  data  set,  prediction  intervals  seem  to  be  well  centered 
and  tight.  This  experience,  while  certainly  inconclusive, 
was  repeated  with  other  data  sets. 

Tables  3-8  are  provided  for  comparing  actual  models 
selected  by  the  various  criteria.  The  selected  models  were 
ranked  according  to  their  frequency  of  occurrence  and  these 
ranks  were  used  for  labeling  purposes.  Maximum  F  was  most 
parsimonious,  selecting  "weight  empty"  as  the  best  predictor 
for  the  cost  of  all  but  two  aircraft.  Minimum  W,  minimum  C., 

2  R 
and  R  were  often  in  agreement  in  selecting  variables  2 ,  5 ,  8 , 

and  12.  As  is  often  the  case,  minimum  MSE  seems  to  overfit, 
using  up  to  eleven  variables  in  eight  different  models. 

Because  MSEP  selected  20  different  models  for  the  23  air¬ 
craft  (none  of  which  were  selected  by  other  criteria) ,  these 
models  are  indicated  by  "x".  Incidently,  the  model  selected 
in  the  document  from  which  the  data  were  obtained  (18 ) , 
was  based  on  variables  4,  6,  and  8,  a  model  not  selected  by 
any  of  our  six  test  criteria. 
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TABLE  3.  Minimum  W  Models  and  Aircraft  Predicted 


A/C 


P-80 

FH-1 

F2H-1 

F7U-1 

F-84E 

F3D-1 

F-86H 

F9F-8 

F4D-1 

F3H-1N 

F-102A 

F-100D 

FJ-4 

F-104A 

FllF-1 

F-105B 

F-101C 

F-106B 

F-4B 

F-5A 

F-4J 

F-111A 

F-8E 


Variables 
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TABLE  5.  Minimum  Ck  Models  and  Aircraft  Predicted 


A/C 


F-80 

FH-1 

F2H-1 

F7U-1 

F-84E 

F3D-1 

F-86H 

F9F-8 

F4D-1 

F3H-1N 

F-102A 

F-100D 

FJ-4 

F-l 04 A 

F11F-1 

F-105B 

F-101C 

F-106B 

F-4B 

F-5A 

F-4J 

F-111A 

F-8E 


Variables 


1 

4  4  4  4 

1 
1 
1 
1 
1 
1 
1 

2  2 

1 
1 
1 
1 
1 
1 
1 
1 

2  2 

1 
1 
1 


1 

4 

1 

1 

1 

1 

1 

1 

1 

2 

1 

1 

1 

1 

1 

1 

1 

1 

2 

1 

1 

1 


4 


k10 


X11  X12 

3 

1 

4 

1 

1 

1 

1 

1 

1 

1 

2 

1 

1 

1 

1 

1 

1 

1 

1 

2 

1 

1 

1 


TABLE  6.  Maximum  F  Models  and  Aircraft  Predicted 


A/C 


Variables 

X1  X2  X3  X4  X5  X6  X7  X8  X9  X10 


F-80 

FH-1 

F2H-1 

F7U-1 

F-84E 

F3D-1 

F-86H 

F9F-8 

F4D-1 

F3H-1N 

F-102A 

F-100D 

FJ-4 

F-104A 

FllF-1 

F-105B 

F-101C 

F-106B 

F-4B 

F-5A 

F-4J 

F-111A 

F-8E 


2  2  2 
1 

3  3  3  3 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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TABLE  8. 

MSEP  Models  and  Aircraft 

Predicted 

A/C 

Variables 

Xx  x2  x3  X4 

X5  x6 

X7 

X8 

X9 

xio 

X11 

F-80 

X  X 

X 

X 

FH-1 

X 

X 

F2H-1 

X 

X 

X 

X 

X  X 

X 

X 

X 

F7U-1 

X 

X 

X 

F-84E 

X 

F3D-1 

X 

F-86H 

X 

X 

X 

F9F-8 

X  X 

X 

F4D-1 

X 

X 

F3H-1N 

X 

X 

F-102A 

X  X 

X 

F-100D 

X 

X 

X 

FJ-4 

X 

X 

X 
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Table  9  indicates  the  importance  of  careful  data  base 
formation  with  respect  to  the  effect  on  prediction  interval 
width  of  including  an  apparently  nonanalogous  aircraft, 
the  F2H-1.  Deleting  this  data  point  decreased  the  average 
width  of  W-optimal  models  by  about  28%  on  the  average. 

In  summary,  the  W-cr iter ion  deserves  close  attention  as 
a  model  building  device  when  the  object  of  analysis  is  pre¬ 
diction  at  a  known  point  in  the  space  of  predictor  variables. 
Mahalanobis  distance  is  a  natural  measure  of  this  analogy 
between  similar  systems.  Ignoring  the  degree  of  analogy 
between  the  object  system  and  the  historical  data  base  can 
result  in  choosing  a  model  which,  in  order  to  predict  the 
cost  of  the  object  system,  may  be  required  to  perform  a  large 
extrapolation,  a  dangerous  practice  in  regression  analysis. 
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TABLE  9.  Observation  Deleted  and  Maximum  Reduction  in 

Width  of  the  Prediction  Interval  for  Each  Aircraft 


A/C 

F-80 

PH-1 

F2H-1 

F7U-1 

F-84E 

F3D-1 

F-86H 

F4D-1 

F3H-1N 

F-102A 

F-100D 

FJ-4 

F-104A 

F11F-1 

F-105B 

F-101C 

F-106B 

F-5A 

F-4J 

F-111A 

F-8E 


A/C 

Deleted 

F2H-1 

F2H-1 

F-102A 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F9F-8 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F2H-1 

F2H-1 


Reduction 

(%) 

33.23 

31.06 

26.01 

25.20 

25.17 

33.22 

16.39 

25.48 
25.59 
25.51 
31.31 
29.53 

7.37 

26.08 

28.79 

25.20 

25.49 
39.37 

32.78 
29.64 

31.78 
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